Does Color Really Help in Dense Stereo Matching

This paper investigates the role of color in global stereo matching approaches. In our evaluation study, we build various energy functions by combining nine color spaces with four dissimilarity functions and test their performance on 30 ground truth stereo pairs. Our experiments start by computing the matching scores via the absolute difference of color values. As is consistent with previous studies, we observe that color-based matching clearly outperforms grey-scale matching. However, our key observation is that this improvement largely stems from considerably improved performance in radiometric distorted regions, i.e. regions where corresponding pixels have different intensities/colors in the two input images, which is e.g. caused by illumination variations. Hence, we claim that color basically serves the same purpose as radiometric insensitive measures, namely to reduce matching errors in radiometric distorted image areas. However, the important difference is that radiometric insensitive measures are considerably superior in this respect, which we demonstrate by using Mutual Information, ZNCC and Census as dissimilarity functions in our experiments. Interestingly, we observe that for these dissimilarity functions color even has a negative effect. Therefore, our suggestion is to not use color at all, but radiometric insensitive measures on grey-scale images, also on images where radiometric distortions seem to be very small.